Apparatus, method, and system of cognitive communication assistant for enhancing ability and efficiency of users communicating comprehension

ABSTRACT

A communication apparatus, a method, a computer readable medium, and a system providing communication with cognitive and visual assistance. The cognitive assistance and visual assistance is provided during a communication between a first communication apparatus with at least one second communication apparatus via a network. The first communication apparatus captures communication data comprising visual and audio information obtained from the communication and captures synchronized cognitive and emotional data generated from the user during the communication with the second communication apparatus. The communication data and the synchronized cognitive and emotional data is stored and converted into a visual form comprising at least one of synchronized text, symbols, sketches, images, and animation. The visual form is displayed on a display of the first communication apparatus.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a Continuation Application of U.S. patentapplication Ser. No. 16/451,326 filed Jun. 25, 2019, which is aContinuation Application of U.S. patent application Ser. No. 16/213,577filed Dec. 7, 2018, which claims priority from U.S. ProvisionalApplication No. 62/668,939 filed May 9, 2018, the contents of which areincorporated herein by reference in their entireties.

BACKGROUND 1. Field

Apparatuses, methods, systems, and computer readable mediums consistentwith the present disclosure broadly relate to cognitive technology, andmore particularly, to cognitive communication related technology.

2. Description of Related Art

Nowadays, global communication is more prevalent with technologicaladvances. Communication using electronic devices is widespread. Usersuse their electronic devices to communicate their daily activities andto interact with friends via social media, online chatting, messagingand multi-media messaging, video and telephone calling, and even playingvideo games or watching media together using their respective electronicdevices.

To accommodate various functionalities required by the users, theelectronic devices are also advancing, for example, by having fasterprocessing, more storage capabilities, improved communicationmechanisms, and improved user interfaces.

However, there are still many unresolved problems in the art ofcommunication technologies. For example, technologies to allow a user tonot only communicate via audio and video but also share his or heractual cognitive state with other users are still needed.

Additionally, there is still a need to develop technologies that couldcapture data of a conversation of a user and display as much as possibleof this data to him or herself to help the user manage his or herworking memory performance during the conversation, to further enhancethe coherence of the conversation. There is also a need to measure anddisplay actual emotional states of a user together with the above dataof the conversation to himself or herself and to his or her friend toenrich the mutual understanding of the information exchanged.

There is yet another need to display the topic of the conversation suchas photos, presentation, on a device of the user such as a mobile phoneto enrich the content of information exchange and to improve thecomprehension of the user during and/or after the conversation. There isyet another need in the art of reviewing, adding notes and comments ontothe recorded video conversation, and then save the added notes and/orexchange them with peers. Additionally, the electronic devices that canaccommodate cognitive assistance during audio and/or visualcommunication while being convenient to use (such as small sized andportable) are need.

In other words, there are many areas in the communication provided byvarious electronic devices that needs further developments andimprovements.

The above information is presented as background to help set the stageof the present disclosure. No acquiescence and no determination has beenmade whether any of the above provided information may be applicable asprior art with regard to the present disclosure.

SUMMARY

Illustrative, non-limiting embodiments may overcome the abovedisadvantages and other disadvantages not described above, and also mayhave been developed to provide solutions to other disadvantages andproblems that were not described above. However, a method, an apparatus,a system, and a computer readable medium that operates according to theteachings of the present disclosure are not necessarily required toovercome any of the particular problems or disadvantages describedabove. It is understood that one or more exemplary embodiment is notrequired to overcome the disadvantages described above, and may notovercome any of the problems described above. The appended claims shouldbe consulted to ascertain the true scope of the present disclosure.

An aspect of the present disclosure is to provide a cognitivecommunication apparatus.

Another aspect of the present disclosure is to provide a communicationmethod in which cognitive assistance is provided.

Yet another aspect of the present disclosure is to provide a system anda computer readable medium which provide cognitive assistance duringcommunication between one or more users.

According to one or more aspects of the present disclosure, acommunication apparatus with cognitive and visual assistance isprovided. The communication apparatus includes at least onecommunication interface configured to connect to at least one network toexecute communication with at least one other communication apparatus,at least one communication data capturer configured to capturecommunication data comprising visual and audio information obtained fromthe communication and further configured to capture synchronizedcognitive and emotional data generated from at least one user during thecommunication with the at least one other communication apparatus, amemory configured to store the communication data and the synchronizedcognitive and emotional data. The communication apparatus furtherincludes at least one communication data converter configured to convertthe communication data and the synchronized cognitive and emotional datainto a visual form comprising at least one of synchronized text,symbols, sketches, images, and animation and at least one displayconfigured to display the visual form of the communication data and thesynchronized cognitive and emotional data converted by the communicationdata converter.

According to one or more aspects of the present disclosure, acommunication method with cognitive and visual assistance is provided.The method includes connecting, by a first communication apparatus, toat least one network to execute communication with at least one secondcommunication apparatus, capturing, by the first communicationapparatus, communication data comprising visual and audio informationobtained from the communication, capturing, by the first communicationapparatus, synchronized cognitive and emotional data generated from atleast one user during the communication with the at least one secondcommunication apparatus, storing, by the first communication apparatus,the communication data and the synchronized cognitive and emotionaldata, converting, by the first communication apparatus, thecommunication data and the synchronized cognitive and emotional datainto a visual form comprising at least one of synchronized text,symbols, sketches, images, and animation, and displaying, on a displayof the first communication apparatus, the visual form of the convertedcommunication data and the converted synchronized cognitive andemotional data.

According to one or more aspects of the present disclosure, anon-transitory computer readable medium is provided. The medium isconfigured to store instructions, which when executed by the processorcause the processor to execute the following operations: connecting afirst communication apparatus to at least one network to executecommunication with at least one second communication apparatus,capturing communication data comprising visual and audio informationobtained from the communication, capturing synchronized cognitive andemotional data generated from at least one user during the communicationwith the at least one second communication apparatus, storing thecommunication data and the synchronized cognitive and emotional data,converting the communication data and the synchronized cognitive andemotional data into a visual form comprising at least one ofsynchronized text, symbols, sketches, images, and animation, anddisplaying, on a display of the first communication apparatus, thevisual form of the converted communication data and the convertedsynchronized cognitive and emotional data.

According to one or more aspects of the present disclosure, a system ofperforming communication with cognitive and visual assistance isprovided. The system includes at least one sensory device, worn on atleast one of a head of a user and a body of the user, which capturessensory signals from the user comprising at least one of brain signals,blood pressure, and skin temperature and a first communication apparatuswhich communicates with a second communication apparatus via a network.The first communication apparatus includes at least one communicationinterface configured to connect to at least one network to executecommunication with at least one other communication apparatus, at leastone communication data capturer configured to capture communication datacomprising visual and audio information obtained from the communicationand further configured to capture synchronized cognitive and emotionaldata generated from at least one user during the communication with theat least one other communication apparatus, a memory configured to storethe communication data and the synchronized cognitive and emotionaldata, at least one communication data converter configured to convertthe communication data and the synchronized cognitive and emotional datainto a visual form comprising at least one of synchronized text,symbols, sketches, images, and animation, and at least one displayconfigured to display the visual form of the communication data and thesynchronized cognitive and emotional data converted by the communicationdata converter.

Additional and/or other aspects will be set forth in part in thedescription which follows and, in part, will be apparent from thedescription, or may be learned by practice of exemplary embodimentsexplained below.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification exemplify embodiments and, together with thedescription, serve to explain and illustrate exemplary embodimentsthereof. Specifically:

FIG. 1 is a block diagram illustrating a configuration of a cognitivecommunication apparatus, according to an exemplary embodiment.

FIG. 2 is a block diagraming illustrating a configuration of acommunication system with cognitive assistance, according to anexemplary embodiment.

FIG. 3A is a view illustrating an apparatus in a closed state, accordingto an exemplary embodiment.

FIG. 3B is a view illustrating an apparatus in a 120-degree open state,according to an exemplary embodiment.

FIGS. 3C and 3D are views illustrating a front side and a back side,respectively, of an apparatus in a fully open state, according to anexemplary embodiment.

FIG. 4 is a block diagram illustrating a hardware configuration of anapparatus providing cognitive assistance for communication according toan exemplary embodiment.

FIGS. 5A-5C are views illustrating various devices which capturecognitive states sensory data, according to exemplary embodiments, andillustrate synchronized cognitive states sensory data captured byvarious devices and interpreted to correspond to a classified cognitivestate of a user, according to exemplary embodiments.

FIG. 6 is a view illustrating an apparatus providing an emotional stateof a user while being in a communication mode, according to an exemplaryembodiment

FIG. 7 is a flowchart illustrating a method of providing an emotionalstate of a user while an apparatus is in a commutation mode according toan exemplary embodiment.

FIG. 8 is a block diagram illustrating components of generating dataaccording to an exemplary embodiment.

FIG. 9 is a view illustrating an apparatus providing secondary contentsof a conversation and an emotion state of a user, while being in acommunication mode, according to an exemplary embodiment.

FIG. 10 is a view illustrating an apparatus providing, in real-time, oneor more scripts or transcripts of a conversation and an emotional stateof a user, while being in a communication mode, according to anexemplary embodiment.

FIG. 11 is a view illustrating an apparatus providing secondary contentsincluding review contents and an emotional state of the user, whilebeing in a review mode, according to an exemplary embodiment.

FIGS. 12A and 12B are views illustrating a front and a back,respectfully, of an apparatus in a fully open state entering acommunication mode according to yet another exemplary embodiment.

FIG. 13 is a view illustrating an apparatus providing an emotional stateof a user while being in a communication mode, according to yet anotherexemplary embodiment.

FIG. 14 is a view illustrating an apparatus providing secondary contentsincluding review contents and an emotional state of the user, whilebeing in a review mode, according to an exemplary embodiment.

FIG. 15 is a view illustrating an apparatus providing primary contents,secondary contents, and an emotional state of the user while theapparatus is in a learning mode, according to yet another exemplaryembodiment.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

Exemplary embodiments will now be described in detail with reference tothe accompanying drawings. Exemplary embodiments may be embodied in manydifferent forms and should not be construed as being limited to theillustrative exemplary embodiments set forth herein. Rather, exemplaryembodiments are provided so that this disclosure will be thorough andcomplete, and will fully convey the illustrative concept to thoseskilled in the art. Also, well-known functions or constructions may beomitted to provide a clear and concise description of exemplaryembodiments. The claims and their equivalents should be consulted toascertain the true scope of an inventive concept. Other technicaladvantages may become readily apparent to one of ordinary skill in theart after review of the following figures and description. It should beunderstood at the outset that, although exemplary embodiments areillustrated in the figures and described below, the principles of thepresent disclosure may be implemented using any number of techniques,whether currently known or not. The present disclosure should in no waybe limited to the exemplary embodiments and techniques illustrated inthe drawings and described below.

According to exemplary, non-limiting embodiments, cognitive and/oremotional assistance is provided to a user during and/or after having aconversation with another user or users. According to an exemplaryembodiment, a conversation may be a telephonic or a video conference.

According to yet another exemplary, non-limiting embodiment, a portableapparatus provides for cognitive or emotional assistance of contentssuch as visual and audio information captured during a conversation withanother user for example. This visual and audio information may then besynchronized with cognitive or emotional states of the user and mayinclude notes and/or further comments.

According to yet another exemplary, non-limiting embodiment, anapparatus is compact and is portable such as a mobile terminal but yetprovides cognitive assistance during and/or after the communication.

For example, U.S. Pat. No. 9,711,056 to Nguyen describes capturing,detecting, and identifying different types of emotional stimulationgenerated by human organs while the human is exploring and observing theenvironment, also incorporated herein by reference for its helpfuldescriptions.

Additionally, U.S. patent application Ser. No. 15/870,210 to Nguyenfiled on Jan. 12, 2018, incorporated herein by reference for its helpfuldescriptions, describes assisting a user in learning, review, andmemorization. '210 application describes capturing, displaying, andprocessing learning data.

However, there is still a need for capturing, displaying, and processingcommunication data. There is a need to provide cognitive assistance asit relates to communication and communication data.

There is a need to build an apparatus, a method, and a system which isconvenient for the user in daily use such that it is portable (easy tocarry) and has mobile communication and data communication available sothat the user may socialize with other users via social networkings,messaging, and video/audio conversations while also providing cognitiveassistance. That is, an electronic device is needed in which the usermay continue with his daily functions and use of an electronic devicewhile having additional cognitive assistance, as described in greaterdetail below in various exemplary embodiments. The provided cognitiveassistance may include capturing the conversation and the emotions stateof the user, display the captured material during conversation andsocial networking, build correlations between the conversation andinsights.

FIG. 1 is a block diagram illustrating a configuration of a cognitivecommunication apparatus, according to an exemplary embodiment.

As shown in FIG. 1, the cognitive communication apparatus 100 includes acontroller 1801, which controls the overall operations of the apparatus100 and/or components of the apparatus 100. The controller 1801 mayinclude at least one processor such as a central processing unit and agraphics processor.

The apparatus 100 also includes a memory 1802. The memory 1802 mayinclude random access memory (RAM) and read only memory (ROM). Thememory 1802 is an internal memory which stores an operating system (OS)for running the cognitive communication apparatus, one or moreapplications, system software, and user data. The memory 1802 mayinclude an external memory such as a memory card (SD card, SIM card,microSD card, and so on).

The cognitive communication apparatus further includes an antenna 1803and a communication interface 1804. The communication interface 1804 isconfigured to receive communications data from another device via anetwork. The communication data may be received via a cellular networkand/or via a data network. The communication interface 1804 may includea network card to provide an Ethernet connection and a wireless LANconnection.

The communication interface 1804 is further configured to receivecognitive data indicating user's emotional and/or cognitive state from acognitive state capturing apparatus, explained in further detail belowwith reference to FIG. 2. The communication interface 1804 may includeone or more of a near field communication (NFC) interface, a Bluetoothcommunication interface, and an infrared communication interface, forcommunicating with the cognitive state capturing apparatus (FIG. 2),which is in a vicinity of the cognitive communication apparatus 100.

Additionally, the cognitive communication apparatus 100 includes adisplay 1805, a microphone 1806 for receiving user input audio data, acamera 1807 for capturing visual data such as the user during thecommunication, speakers 1808 for outputting audio component of thereceived communication data, and a power source 1809. Additionally, thecognitive communication apparatus 100 may include an HDMI adapter, apower source outlet, a PC port, and a USB port, by way of an example andnot by way of a limitation, as is known in the art.

The display 1805 of the cognitive communication apparatus 100 may be atouch display and may include a number of sensors to capture user input.This is provided by way of an example and not by way of a limitation.The cognitive communication apparatus 100 may be connected to a keyboardand/or other user input interfaces via one or more of the USB ports (notshown).

A cognitive communication apparatus 100 may be a personal computer (PC),a notebook computer, a smart telephone, and so on. These are provided byway of an example and not by way of a limitation.

In order for the cognitive communication apparatus 100 to providecognitive assistance to the user, the user's cognitive state iscaptured. FIG. 2 is a block diagram illustrating a configuration of acommunication system with cognitive assistance, according to anexemplary embodiment.

As shown in FIG. 2, the cognitive communication apparatus 100communicates with a cognitive state capturing apparatus 1900 via anetwork 1901 to obtain at least a portion of a cognitive state of theuser. The cognitive state capturing apparatus 1900, according to anexemplary embodiment, is described in further detail with reference toFIG. 5A. According to an exemplary embodiment, the cognitive statecapturing apparatus 1900 includes a number of sensors to capturecognitive data generated from or by the user and to provide the capturedcognitive data to the cognitive communication apparatus 100, asexplained in greater detail below. The captured cognitive data may becommunicated from the cognitive state capturing apparatus 1900 via ashort range communication network such as a Bluetooth network. Thecognitive data is analyzed to determine the cognitive state of the userand a level of the cognitive state of the user, as explained in greaterdetail below.

The cognitive communication apparatus 100 also transmits and receivescommunication data from another user (user 2) via the user device 1903using a network 1902. The network 1902 may be a data network, accordingto an exemplary embodiment. The communication data (visual and audiodata) may be recorded by the cognitive state capturing apparatus 1900 orby the cognitive communication apparatus 100.

According to various exemplary embodiments, at least some of thecomponents described to be part of the cognitive communication apparatus100 may be included in the cognitive state capturing apparatus 1900 andvice versa, as would be readily apparent to one of ordinary skill in theart based on various exemplary embodiments. Additional remote serversmay be provided for data analysis, according to an exemplary embodiment,such that the cognitive state capturing apparatus 1900 may be in directcommunication with the server or via the cognitive communicationapparatus 100.

The cognitive communication apparatus 100 may include only one displaywhere various data is displayed thereon or a plurality of displays. Inan exemplary embodiment, explained in further detail below, a dualdisplay apparatus is described but this is provided by way of an exampleonly to help understand exemplary embodiments of the present disclosureand not by way of a limitation.

FIGS. 3A-3D are views illustrating an apparatus according to one or moreexemplary embodiments. The apparatus depicted in the figures is acognitive or an emotion detection enabled, mobile, portable device suchas a dual-display smart phone. FIG. 3A is a view depicting an apparatus100 in a closed state. The apparatus 100 has two parts: a first part 101and a second part 102. The first part 101 and the second part 102 have aprotective casing on an outer part thereof as depicted in FIG. 3A. Nocameras or other components of the apparatus 100 extend to the outside,according to an exemplary embodiment. The first part 101 may have avolume button 103 and a lock state button 104, as would be understood byone of ordinary skill in the art. Additionally, buttons 105 and 106 maybe provided on the second part 102 to open the apparatus 100 and tosilence the apparatus 100 and a recessed hole 107 may be provided forcapturing audio environment. These features are provided by way of anexample and are not intended to limit the scope of the presentdisclosure. According to an exemplary embodiment, since no camera isprovided on an outer portion of the parts 101 and 102, sturdiness of theapparatus is improved and complexity of the apparatus is reduced.

The apparatus in its closed state, as depicted in FIG. 3A, is compact tobe easily carried by the user. For example, the apparatus in its closedstate maybe 4 to 10 inches in length, 2 to 6 inches wide and 0.3-2inches in depth. These ranges in size are provided by way of an exampleand not by way of a limitation. Accordingly, the apparatus may be easilycarried by a user in a pocket or a purse and is compatible with variousdevice holders, known in the art.

FIG. 3B is a view illustrating an apparatus in a 120 degree open stateaccording to an exemplary embodiment. In FIG. 3B, the apparatus 100 hasthe first part 101 and the second part 102, each of which includes adisplay. Specifically, the apparatus 100, according to an exampleembodiment, has a first display 201 and a second display 202. Accordingto an exemplary embodiment, both displays 201 and 202 may be touchscreen displays but this is provided by way of an example and not by wayof a limitation. According to other exemplary embodiments, only one ofthe displays may be a touch screen display or none of the screendisplays may be a touch screen display. If none of the displays are atouch screen display, the apparatus may be operated using an add-onkeyboard or using keys, joysticks, and other controls as is known in theart.

According to an exemplary embodiment, the display 201 is smaller in sizethan the display 202. According to yet another exemplary embodiment, thedisplay 202 may be smaller in size than the display 201. One of thedisplays is made smaller to provide space for a camera 203 and otherhardware components 204, according to an exemplary embodiment. Thecamera 203 allows for selfies and video conferences (video chats) whenthe apparatus is open 120 degrees. The camera 203 allows the user tocapture the environment in a different state, as described in furtherdetail below. The hardware components 204 may include a functionalbutton or a place to return to a home screen in an apparatus, a backbutton to return to the previous screen, or a display switch button suchthat contents displayed on the display 201 are switched with contentsdisplayed on the display 202. These explanations are provided by way ofan example and not by way of a limitation.

FIG. 3B shows the captured images and/or video and/or other contents(hereinafter referred to as “primary contents”) being displayed on afirst portion 205 of the first display 201. Additionally, the firstdisplay 201 may display a time line 206 which corresponds to thecontents displayed in the first portion 205. Additionally, a secondportion 207 of the first display 201 may provide a quick key to jump tovarious portions within the primary contents displayed in the firstportion 205. The quick keys may vary according to size and indexing ofthe primary contents or may be custom partitioned or set by the user. Inan exemplary embodiment, the captured contents is part of communicationdata obtained in real time, which may be reviewed at a later time andadditional text input maybe provided in the obtained communication data,as explained in further detail below.

Additional contents (hereinafter referred to as “secondary contents”)related to the primary contents may be provided on a second display 202.This is provided by way of an example and not by way of a limitation.The second display 202 may display scripts (converted text of theprimary contents) which are synchronized in real time with actualportions of the primary contents being displayed. Additionally, thesecond display 202 may display notes related to the primary contents aspreviously captured by the user and/or other users or is currently addedby the user and/or other users. The second display 202 may providefunctionalities available with respect to the secondary contents such assearch, edit, delete, save, and add. The second display 202 may alsoprovide icons related to other applications available in the apparatus100, explained in further detail below according to various exemplaryembodiments. The second display 202 may further provide a color-codedtimeline 207 in which cognitive data (user's emotions) are color codedwith respect to time.

Additionally, according to an exemplary embodiment, captured cognitiveor emotional state of a user of the apparatus (hereinafter referred to“cognitive contents”) may also be provided on the first display 201 butis not limited thereto. The cognitive or emotional state of the user maybe provided on the second display 202 or on both displays 201 and 202 invarious forms, as described in greater detail below according to variousexemplary embodiments.

FIGS. 3C and 3D are views illustrating the apparatus 100 in a fully openstate (360 degree open) according to an exemplary embodiment. In thisopen state, the front side is the second part 102 and may display acamera icon 301 (shown in FIG. 3C) to capture the environment with thecamera 302 (shown in FIG. 3D). In an exemplary embodiment, icons ofvarious functions and/or applications executable by the apparatus may bedisplayed on the first part 101 (which is referred to as a back display303, shown in FIG. 3D) or on the front display of the second part 102(shown in FIG. 3C). FIG. 3C shows the front display (second part 102)displaying that the apparatus is entering a communication mode in whicha telephone call or a video call is being made to another user(“Susan”), according to an exemplary embodiment. The communication modeis explained in further detail below according to various exemplaryembodiments. When the apparatus is open 360 degrees (FIGS. 3C and 3D),the device may be working the same as other smart phones known in theart. The camera 302 is on an internal side of the first part 101 andfaces the environment when the apparatus is in an open 360 degrees stateand an internal side of the second part 102 faces the user. If the userwant to take a photo, he or she just touches an icon displayed on theinternal side of the second part 102 in an analogous ways as using othersmart phones and is known in the art, according to an exemplaryembodiment. In an exemplary embodiment, the environment of the user maybe captured with the camera 302 and/or shared during a communicationusing the first part 101, as explained in greater detail below accordingto various exemplary embodiments.

According to an exemplary embodiment, in the 360 degree open state ofthe apparatus 100, the internal side of the second part 102 becomes thefront side that faces the user and that is being observed andmanipulated by the user and the internal side of the first part 101becomes the back side that faces the environment of the user to captureimages and/or video of the environment, provide audio and visual outputto the environment (such as a flashlight functionality). These areprovided by way of an example and not by way of a limitation. Accordingto an exemplary embodiment, the user may flip the phone so that thefirst part 101 faces the user and select some of the applicationsavailable on the first part 101.

FIG. 3C shows that the user of the apparatus, Mary, is making a call toher friend Susan. FIG. 3C shows the apparatus 100 entering acommunication mode, as explained in further detail below according tovarious exemplary embodiments.

FIG. 4 is a block diagram illustrating a hardware configuration of anapparatus providing cognitive assistance for communication according toan exemplary embodiment.

In FIG. 4, the apparatus 100 has a controller 410, a first display 420and a second display 430, communication interfaces 440, a memory 450,audio interfaces 460, sensors 470, and connectors 480. These areprovided by way of an example only and not by way of a limitation. Oneof ordinary skill in the art would readily appreciate that a smart phonemay include a variety of these hardware components and other hardwarecomponents in various combinations.

The controller 410 may include at least one processor 411 including twographical processors (GPU). One GPU may be provided for each display.The one or more processor 411 may further include a central processingunit (CPU), a video processor, an audio processor, a random accessmemory (RAM) 412 which stores signals external to the apparatus 100 orvarious instructions and operations to be executed by the apparatus 100,a read-only memory (ROM) 412 in which a control program such as anoperating system software is stored to control functions of the displayapparatus. The components of the controller 410 are connected to eachother via an internal bus 414. The controller 410 controls thecomponents of the apparatus 100. The controller may further includeinterfaces (not shown) to communicate with various internal components.Additionally, at least some of the components explained above may beprovided on a separate chip and not be a part of the controller. Forexample, a video and/or audio processor may be provided as separatehardware components.

The first display 420 and a second display 430 may display first,second, and cognitive contents, as explained in further detail below.Additionally the displays 420 and 430 may be a touch screen such as toreceive user input via touch.

The communication interfaces 440 may receive and transmit signals and/ordata from the outside (including external devices and/or servers). Forexample, the communication interfaces 440 may include a network card andother hardware components in combination with software that provide fora wireless local rear network (WLAN) communication, Bluetoothcommunication, short-range communication, cellular communication, anddata network communication, as may readily be understood by one versedin the art. For example, when the apparatus 100 enters into thecommunication mode, signals from one or more other apparatus arereceived via one or more of the communication interfaces 440.

A memory 450 may include an internal memory of the apparatus 100, whichmay store primary, secondary, and cognitive contents. The memory 450 mayinclude a hard disk, a memory card, a secure digital card, a smart cardsuch as a SIM card, and so on. These are provided by way of an exampleand not by way of a limitation. Additionally, the apparatus 100 maycommunicate with a cloud server (not shown) and obtain contents from thecloud server.

The audio interfaces 460 may include one or more audio output interfacessuch as one or more speakers and a headphone socket to provide an audiooutput to headphones. The audio interfaces 460 may further include oneor more audio input interfaces such as a microphone. These are providedby way of an example and not by way of a limitation. For example, duringthe communication of the apparatus 100, audio data may be captured viathe audio interfaces 460 and may also be output via the audio interfaces460 under the control of the controller 410.

The sensors 470 may include a camera and one or more of optical sensors,distance and pressure sensors such as the ones that capture change inthe flow of electricity on a touch screen, and motion sensors, as wouldbe readily appreciated by one of ordinary skill in the art. These areprovided by way of an example and not by way of a limitation.

The connectors 480 may include a USB port connector for communicatingdata to and from another device, a power supply connector, and so on, aswould be readily apparent to one versed in the art. These are providedby way of an example and not by way of a limitation.

FIGS. 5A-5C are views illustrating a device which captures supportingcommunication contents and sensory data including cognitive andemotional data, according to an exemplary embodiment, and illustratingcognitive and emotional data captured by the device and interpreted tocorrespond to a classified cognitive and emotional state data of a user,according to an exemplary embodiment.

In an exemplary embodiment, two types of memory that the brain isworking with during communication are imitated. Type one is adeclarative memory or an explicit memory related to events, facts,thoughts, people, objects, and places that are addressed during aconversation. That is, elements or objects that are discussed or come upin a communication. Type two is a non-declarative memory, proceduralmemory, or implicit memory related to skills, habits, and naturalreactions in a communication. That is, emotions or reactions to objectsor elements that comes up during the communication.

In an exemplary embodiment, cognitive data is captured from thebrainwave or other physiological signals generated when the brain isworking with the type one memory (explicit memory). Additionally,emotional data is captured from brainwave or other physiological signalsgenerated when the brain is working with the type two memory i.e.,implicit memory.

As illustrated in FIG. 5A, a camera 501, which includes a microphone,may be provided on a front portion of the headset 500, according to anexemplary embodiment. This is provided by way of an example and not byway of a limitation. Multiple cameras may be provided such as a leftcamera, a front camera, a right camera, and a back camera to capturevisual data and/or audio data according to an exemplary embodiment. Oneof ordinary skill in the art would readily appreciate that visual dataand/or audio data may be captured with a personal device such as auser's personal data assistant or a cellular telephone. The capturedvisual and audio data (VI) may then be transferred to an electroniccircuit board 502, which includes at least a memory coupled with aprocessor.

In an exemplary embodiment, the electronic circuit board 502 may processsensory data to generate cognitive and emotional state of a user. In yetanother exemplary embodiment, the generated cognitive and emotionalstate information may be transmitted to another remote device forstorage, monitoring, or further processing via a communication interface(not shown) provided on the headset 500. For example, the headset 500may include a communication interface (e.g., a network card, an antenna,and other interfaces known to one of ordinary skill in the art or laterdeveloped) to transmit the data wirelessly e.g., a Bluetooth, Infrared,WiFi, and/or a cellular network to a remote server or a cloud forfurther storage, processing or monitoring and co-supervising. Thecommunication interface may be built into the electronic circuit board502, may be built into the communication device 100, or may be providedas a separate device on the headset 500. According to an exemplaryembodiment, a USB port may be provided on the electronic circuit board502 or separately on the headset 500 so as to plug into a computer totransfer captured data (VI data and sensory data).

In an exemplary embodiment, one or more emotional sensors or cognitivestate sensors 503 are further provided on a headset 500. While FIG. 5Adepicts four cognitive state sensors 503, this is provided by way of anexample and not by way of a limitation. One of ordinary skill in the artwould readily appreciate that a single sensory or cognitive state sensormay be used but preferably multiple cognitive state sensors are providedto capture cognitive state of a user. The cognitive state sensors 503may be provided on both sides of the headset 500. In an exemplaryembodiment depicted in FIG. 5A, only one side of the user's head isshown but the other side may also include four cognitive state sensors503 that detect the cognitive state of the user. That is, in anexemplary embodiment, cognitive state is obtained from multiple sensors503 by detecting activities in various parts of the brain.

FIG. 5B is a view illustrating another apparatus which capturessupporting communication contents and sensory data including cognitiveand emotional data, according to an exemplary embodiment. As shown inFIG. 5B, a device 510 may be worn on a finger of a user and captureemotional states of the user during a conversation, according to anexemplary embodiment. The device 510 captures emotional states of theuser through physiological signals by measuring Heart Rate (HR), by wayof an example.

According to an exemplary embodiment, FIG. 5B may also include anotherdevice 520 worn on the hand of the user similar to a watch. This otherdevice 520 captures emotional states of the user during a conversation.The another device 520 captures emotional states of the user throughphysiological signals by measuring items like blood volume pulse (BVP),skin temperature (SKT), by way of an example and not by way of alimitation.

According to an exemplary embodiment, EEG and other physiologicalsignals such as HR, BVP, SKT are provided by way of an example and notby way of a limitation.

According to an exemplary embodiment depicted in FIG. 5B, the user maybe using the apparatus 100 with his or her hands while wearing thedevice 500. For example, as explained in further detail below, the usermay be looking in front of her while walking on a path in an alley withSakura trees on both sides of the alley, as shown in FIG. 6. The usermay be talking with friend (Susan) about the event which is livestreaming with her devices.

According to an exemplary embodiment, as shown in FIG. 5B, the user isusing apparatus 100 while wearing at least one of the devices 500, 510,and 520. According to an exemplary embodiment, explained below withreference to FIG. 6, the user may be looking in front of him or her atthe device, which displays people walking in an alley in a park. Theuser may be talking to a friend about the event which is live streamingwith her devices. The devices 500, 510, and 520 will capturephysiological data, according to an exemplary embodiment.

According to an exemplary embodiment, raw sensory signals obtained fromthe sensors are combined and synchronized with the communication. In anexemplary embodiment, the raw signals embody sensory distinctiveinstinct components. In an exemplary embodiment, raw sensory signals arefiltered at distinctive reference signals to detect ET1 component forthe levels of loving state where a user is instinctively engaged andattracted (excited, loved) during the conversation, ET2 component forthe levels of liking state where a user is consciously (orrationally—please recommend which one is used better) engaged andpersuaded (liking) during the conversation, and ET3 component for thebored state where a user is feeling bored or not-engaged during theconversation, as shown in FIG. 5C and explained in greater detail below.

According to an exemplary embodiment, ET1 indicates the emotional statesat a very high level, ET2 indicates both emotional and cognitive statesat rather high levels, ET3 indicates a low level of both emotional andcognitive states, as shown in FIG. 5C.

According to an exemplary embodiment, when the discrimination betweencertain sensory signals' components and referenced signals are increasedor decreased means that the user is at an identified level of anidentified state such as excited (very interested, naturally attractedand engaged), liking (rationally engaged and persuaded, good feeling,like), and so on. For some examples, see U.S. Pat. No. 9,711,056 andU.S. patent application Ser. No. 15/870,210 filed on Jan. 12, 2018,which are incorporated herein by reference in their entireties.

According to an exemplary embodiment, the cognitive state sensors 503may be positioned to detect levels of main emotions of the user such asliking, loving and exciting, or boring. For example, as shown in FIG. 5Aand 5B, four sensors 503 (S1-S4) are placed on head of the user andoutput respective sensory signals ss1-ss4 and one emotional sensor S5 isplaced on the other part of the body of a user such as an apparatus 510and/or 520. In an exemplary embodiment, if the ET1 signals detected fromsensors S5 output respective signals above predetermined respectivevalues, a determination can be made that the user feels very excitede.g., the user is highly engaged in the communication and thecommunication is stimulating instinctive interests to the user. If thedetected ET2 signals from sensors S1, S3, output respective signalsabove predetermined respective values, a determination can be made thatthe user feel that he is engaged and liking or feeling good about theconversation. If the detected ET 3 signals from sensors S2, S4, outputrespective signals under predetermined respective values, adetermination can be made that the user feels that he is not engaged andbored with the conversation.

In an exemplary embodiment, because the user reads text from top tobottom (i.e., from the upper line to the lower line), therefore signalsof voice which are interpreted synchronously with the text lines. Assuch, in an exemplary embodiment depicted in FIG. 5C, audio signalsi.e., voice output, is displayed in a vertical line(s) and the timelineis also displayed in a vertical direction, for consistency with thetext.

FIG. 5C, according to an exemplary embodiment, depicts voice signalscaptured for an episode of 30 seconds. For example, a timeline 504 isdepicted on the left side of FIG. 5C and is split into 13 intervalsdepicted by 2.5 seconds for each interval. The 5 seconds mark depictsvoice data already spoken by the user and/or the other user of theconversation and the −5 seconds mark depicts voice data to be spoken bythe user and/or the other user of the conversation in 5 seconds (if theconversation has been recorded and is being played back by the user). 0seconds mark indicates the current position of the voice data beingoutput. According to an exemplary embodiment, 0 seconds indicate thevoice data and visual data currently being presented during a videoconference or a teleconference between a user and another user.

According to an exemplary embodiment, the first voice A of a first usermay be captured during time intervals V1-V3, a second voice B of anotheruser may be captured during time intervals V3-V8, and the third voice Cof the first user may be captured during the time intervals V9-V13.These are provided by way of an example only and not by way of alimitation. The captured voices are split into voice segments. Based onthe split voice modules or segments V1, V2, . . . V13, voice to textconversion is performed. That is, voice data is converted into text dataand may be displayed to the user, as explained in greater detail below.In an exemplary embodiment, one text box is provided for each voicesegment or module. For example, as depicted in FIG. 5C, the text,obtained from converting the voice signal, is displayed in respectivetext boxes TB1-TB13. That is, the first text corresponding to theconverted voice segment V1, is displayed in the text box TB1; the secondtext corresponding to the converted voice segment V2, is displayed inthe text box TB2; and the 12^(th) text corresponding to the convertedvoice segment V12, is displayed in the text box TB12.

In an exemplary embodiment, the text output corresponding to theconverted voice is placed into 13 corresponding script lines, which areTB1, TB2, TB13 and script (−15 _(S)), script (−12.5 _(S)) . . . script(15 _(S)) depicted in FIG. 5C. In an exemplary embodiment, the number ofsub-windows being displayed will correspond to the number of voicesegments generated during the division of continuous voice output. Asshown in FIG. 5C, cognitive state of the user is also displayed as fiveemotional state signals ss1-ss5 obtained from the sensors S1, S2 . . .S5. These cognitive state signals are also displayed synchronized withvoice signal in the vertical direction. According to an exemplaryembodiment, the cognitive state signals ss1-ss5 are also split intosegments corresponding to the voice segments V1-V13. That is, in anexemplary embodiment, the emotional state signals are analyzed insegments which correspond to the voice segments to determine anemotional state of the user.

As shown in FIG. 5C, E1 is an emotional state obtained from analyzing ofat least one of signals ss1, ss2, ss3, ss4, and ss5 produced at time 15sec to 10 sec and corresponding to the voice segment V1 to V4, the textof which is displayed in the TB1 to TB4 box. In an exemplary embodiment,E3 is interpreted to correspond to an emotional state 3 and can bemarked with a first color (e.g., blue) to indicate that the user isbored, not engaged (or neutral emotion) about the conversationcorresponding to the voice segment V8 to V13 . That is, E3 indicates astate in which the user is feeling bored about the contents of the voicesegment V8-V13. E2 is the cognitive state obtained from at least one ofsignals ss1-ss5 recorded during the conversation from V5 to V7 withcontent displayed in TB5 to TB7. E2 is interpreted to correspond to anemotional state 2 and can be marked with a second color (e.g., green) toindicate that the user is enjoying or liking the contents presented(voice segment V5-V7) and visual contents). E1 is the emotional stateobtained from at least one of signals ss1-ss5 recorded during the timethe user hears voice V1 to V4 with content displayed in TB1 to TB4. E1is interpreted to correspond to an emotional state 1 and can be markedwith a third color (e.g., red) to indicate that the user is excitedabout the contents (voice segment V1 to V4 and visual contents). Themarking depicted in FIG. 5C are provided by way of an example and not byway of a limitation. According to another exemplary embodiment, the textin the respective text boxes TB1-TB13 can be color coded based on thedetermined emotional state (E1-E3) of the user.

As an alternative, various marking and indicators could be used todepict the user's state including % level of each color, mixing 3 colorsR (red), G (green), and B (blue) to reflect a general emotional state ofthe user during the conversation in real time, as explained in furtherdetail with reference to FIGS. 6 through 13.

According to an exemplary embodiment, the voice signal 506 may be avoice of the user and/or another user during a teleconference, a videoconference, or a gaming session, for example. Contents shared during thevideo conference maybe recorded as visual data and stored insynchronization with the audio signals output by the users, for example.

According to an exemplary embodiment, the memory of the electroniccircuit board 502 may store captured primary data (including thecaptured video data and audio data) and the captured sensory data, whichmay be the EEG or other physiological signals, explained above. Theprocessor of the electronic circuit board 502 may execute asynchronizer, which is configured to synchronize the captured primarydata with the captured sensory data, by way of an example. At thispoint, the processor of the electronic circuit board 502 may transmitthe synchronized data modules including the primary data with thecorresponding sensory data to another processor or a server for furtherprocessing via the communication interface of the electronic circuitboard 502. According to another exemplary embodiment, the processor ofthe electronic circuit board 502 may execute an emotional statedeterminer, which determines the emotional state of the user ET1-ET3.According to an exemplary embodiment, the processor of the electroniccircuit board 502 may determine the emotional state of the user based onthe sensory data provided by the emotional state signals SS1-SS5 at timet0 corresponding to the primary data at time t0. Additionally, theprocessor may convert an audio part of the primary content (user voicesduring the teleconference or videoconference) to text via a speech totext converter executed by the processor of the electronic circuit board502 or executed by a remote server or computer. Next, a scriptgenerator, executed by the processor of the electronic circuit board,generates scripts of the audio contents. Additionally, the scriptgenerator may generate a plurality of text boxes for the segmentedscripts and color code the text boxes (by way of an example) based onthe sensory data using the output from the synchronizer, the emotionalstate determiner, and the speech to text converter. The primarycontents, along with the emotional states may be output to a user viaone of the displays of the apparatus 100.

As shown in FIG. 3C, discussed above, the apparatus 100 is entering intoa video conference mode according to an exemplary embodiment. As shownin FIG. 3C, a user named Mary is calling her friend Susan. Mary iswearing I SEE® headset (the headset 500) that detects and monitors heremotional and/or cognitive state, which she may view herself and/ordecide to share with her friend Susan via one of the displays of herapparatus 100, as explained in further detail below.

FIG. 6 is a view illustrating the apparatus 100 in a communication mode,according to an exemplary embodiment. As shown in FIG. 6, Mary and Susanare using video chat executed via a selection of an icon 601. Mary istraveling to see the Sakura trees in Japan in spring and livestreamchatting with Susan about what she is seeing and feeling in Japan. Thatis, when Mary selects an icon 601, as is known in the art or laterdiscovered, the apparatus 100 transitions into the communication mode inwhich a video conference is conducted. According to an exemplaryembodiment, the apparatus enters into a communication mode once aconnection is established with another user, Susan, as shown in FIG. 6.

According to an exemplary embodiment, the apparatus 100 may record audioand visual data of the communication such that the headset 500 capturesthe cognitive or sensory data and also the visual and audio of theenvironment that user, Mary, is in. The communication data may besynchronized with sensory data via a world-time clock signal forexample. According to another exemplary embodiment, the headset 500 onlycaptures the user's emotional and cognitive data in addition to thesensory data and other components of the apparatus 100 captures thevisual and audio of the environment such as a camera 203.

As shown in FIG. 6, the first part 101 of the apparatus 100 has a firstdisplay 201 and a camera 203. The camera 203 is configured to captureMary in real time during her communication with Susan and is shown asenable on the first display 201. According to an exemplary embodiment,the camera 203 provides video conferencing functionality to display theprimary contents (video and audio captured by the camera 203 and/ormicrophone). The display 201 is split into two portions, a first screen602 and a second screen 603. The first screen 602 displays Susan inreal-time and the second screen 603 depicts Mary, the user of theapparatus 100, in real time. Mary is wearing the I SEE® headset 500 todetermine her emotional or cognitive state and capture video of what sheis seeing and talking about. In an exemplary embodiment depicted in FIG.6, a video chat is conducted between Mary (hereinafter referred to as auser), and Susan (hereinafter referred to other entity). These areprovided by way of an example and not by way of a limitation.

According to an exemplary embodiment depicted in FIG. 6, Mary is tellinga live event to Susan. The visual and audio data captured by theapparatus 100 is referred to as primary data (communication data) and isdisplayed on the first display 201. As is known in the art, the videoand/or audio data of the user is shared with the other entity, Susan.

The visual and audio data captured by the headset 500 is referred to aslive streaming data (live environment data) and is displayed on thesecond display 202. As is known in the art, the video and/or audio dataof the user is shared with the other entity, Susan. In an exemplaryembodiment, in addition to viewing Mary during the video chat, Susan isalso provided with Mary's environment i.e., the path in the park onwhich Mary is currently walking.

Mary also may wish to show additional contents (such as other images,videos, websites, photos, text documents, and so on). Accordingly, Marymay select one of the icons displayed on the second display 202 of thesecond part 102 of the apparatus 100 to share additional contents.

As shown in FIG. 6, other applications may be accessed from the videoconference according to an exemplary embodiment. For example, the usermay select an icon 604 to obtain contents from Internet for the sharingduring the video conference. The user may select an icon 605 to obtainvideo contents from a YouTube (video Internet source) for the sharing.The user may select a gallery icon 606 to obtain images and/or videosstored by the apparatus 100 for the sharing. The selected image will bedisplayed in a third screen 608 of the second display 202 indicated asshared. That is, according to an exemplary embodiment, the third screen608 is displayed by the apparatus 100 to the user as the contentselected and shared with the other entity, Susan. As shown in FIG. 6,the gallery icon 606 is selected and the gallery application is executedfrom the video chat 601 and the additional image selected via thegallery application is displayed in the third screen 608, as beingshared by the user. These are provided by way of an example only and notby way of a limitation.

According to an exemplary embodiment, the third screen 608 may displaycurrent video contents captured by the device 500 (Mary's liveenvironment data) or may be replaced by an image selected from a gallery606, for example. According to yet another exemplary embodiment, thethird screen 608 may be split into two sub screens such that first subscreen plays Mary's live environment data and a second sub screendepicts additional contents selected via various applications availableto the display apparatus.

As shown in FIG. 6, the video chat icon 601 may be shown as enabled toindicate that a video conference is currently occurring. Additionally,the gallery icon 606 is shown as enable to indicate that the additionalimage displayed in the third screen 608 and shared during the videoconference is from the enabled gallery application.

According to an exemplary embodiment depicted in FIG. 6, the user, Mary,may also select a FUVI icon 607 while in the apparatus 100 is in thecommunication mode. That is, the user may select the Fuvi icon 607 fromwithin conducting the video chat 601. According to an exemplaryembodiment, when the Fuvi icon 607 is enabled, the Fuvi icon isindicated as enabled on the second display 102 and a corresponding sixthsense application is executed to capture the emotional state of theuser. The sixth sense application executed by the apparatus 100 sends astart signal to the headset 500 indicating that the headset 500 shouldcapture the live streaming video (live environment data) and thecorresponding sensory data. In response to receiving a start signal fromthe apparatus 100, the headset 500 begins capturing environment data insynchronization with the emotional sensory data.

According to an exemplary embodiment, the headset 500 may broadcast thecaptured live environment video and sensory data in real time with theapparatus 100 and the sixth sense application being executed on theapparatus 100 may execute the emotional state determiner, the speech totext converter, and the script generator, explained above with referenceto FIGS. 5A-5C. According to yet another exemplary embodiment, theheadset 500 may execute the state determiner, the speech to textconverter, and the script generator. According to yet another exemplaryembodiment, the headset 500 may communicate with an external server toobtain the text blocks and the emotional state of the user andcommunicate the same to the apparatus 100. The sixth sense applicationthen displays the emotional state 610 of the user, Mary, in real timeduring the video communication.

That is, according to an exemplary embodiment, the second screen 603 maybe color coded in real time based on current cognitive or emotionalstate of the user (Mary) as captured by the headset 500. A cognitive mapor key of emotional states 609 may also be displayed in the secondscreen 603, color-coded to show various emotions. This is provided byway of an example and not by way of a limitation. According to yetanother exemplary embodiment, not to obscure the third screen 603, theemotional state of the user may be displayed color coded and as text 610in the third screen 603.

According to an exemplary embodiment, the emotional state of the usermay be shared with the other entity. That is, the user may decide toshare her emotional state with the other entity. As such, the otherentity, Susan, not only obtains the video of Mary in real time and theadditional contents being shared via the third screen 608 but alsoMary's actual emotional state 610. That is, although the user may besmiling, the other entity would know that the user is feeling sad orupset and the smile is a fake smile based on the emotional statecaptured by the headset 500. Similarly, even if the emotional state ofthe user is not being shared, the user herself may realize her emotionalstate. For example, the user may be sharing a memory with Susan sayinghow much she hated her trip to the park but her emotional state mayactually indicate that she liked the trip to the park. As such, the userknows her actual state as the memory or story is being shared.

According to an exemplary embodiment, the other entity, Susan, mayunderstand the user of the apparatus not only based on audio/video/andadditional contents being shared during the video conference but alsobased on the cognitive or emotional state of the user. For example,Susan may show the user a new dress she just purchased as additionalcontents in the third screen 608 and as the user is saying how cute thedress is, the other entity may also see the user's emotional state aslike and know, with certainty, that the user actually likes the dressand is not just saying this. As such, the user and the other entity maycommunicate not only in real time with additional contents being sharedbut by also sharing the emotional state of the user to enhance thecommunication. The emotional states are a spectrum, as shown in thecolor coded key 609, where green is like, yellow is between like andexcited, red is excited, purple is dislike, and blue is relaxed.

In an exemplary embodiment, I SEE® headset 500 is used to detect(filter) emotional data and divide it into 3 components (from 3different bands), according to an exemplary embodiment. Three bands areprovided by way of an example and not by way of a limitation. Accordingto an exemplary embodiment, a first band is represented by R (red color%), a second band is represented by G (green color %), and a third bandis represented by B (blue color %). By mixing these 3 colors with itsreal-time percentage, different colors are obtained, which reflectuser's emotional states in real-time. In an exemplary embodiment, thegreen color (third band) relates to “like” state of the user. That is,the user likes what he or she is seeing (pleasant emotions). The redcolor (first band) represent “exciting” state and may relate to “love”.The blue color (second band) relates to a “relaxed” state. As a result,for example, a purple color (which would be a combination of blue andgreen, a predetermined percentage of the two) may be related to boredand/or sleepy. In an exemplary embodiment, mixed percentages of thesethree bands represent a different state of the user analogously tomixing various colors. In an exemplary embodiment, the principle issimilar to color analyzing and processing for digital coloring.

FIG. 7 is a flowchart illustrating a method of providing an emotionalstate of a user while an apparatus is in a communication mode accordingto an exemplary embodiment.

In operation 701, the apparatus enters a communication mode. Accordingto an exemplary embodiment, communication between a user of theapparatus and another entity is established e.g., a video conference, avirtual gaming session, or a teleconference. In operation 702, the firstdisplay of the apparatus displays the user in real-time and the otherentity in real time, and the second display of the apparatus displaysenvironment data in real time and various applications that may beselected while the apparatus is in communication mode. In an exemplaryembodiment, the environmental data is video/audio data captured by thedevice 500 in real time. In an exemplary embodiment, the environmentaldata is environment observed by the user wearing the headset 500. Inoperation 703, the apparatus determines if a user selects one of theicons 601, 604, 605, and 606 to select additional contents to bedisplayed on the third screen 608 of the second display 102. If no, themethod remains at operation 702 where the user and the entity aredisplayed in real time and the camera 203 and the video chat icon 601are displayed as enabled. If the user selects one of the icons (Yes inoperation 703), a list of available additional contents is provided tothe user for a selection. That is, a source of the additional contentsis selected in operation 703, a list, thumbnails, a search field, andother fields, as are known in the art, are provided to the user for aselection of additional content, in operation 704. In response to a userselecting one of the contents in operation 705, the selected content isdisplayed on the third screen 608 and is shared via a communicator withanother apparatus of the other entity in the operation 706. Accordingly,as shown in FIG. 6, the user Mary and the other entity Susan aredisplayed on the first display 201 and the selected additional contentis displayed on the second display 102. According to yet anotherexemplary embodiment, the content selected in operation 705 may bedisplayed on the third screen in operation 706 and may only be sharedupon user confirmation. That is, the selected contents may be availablefor the user only and not shared with the other entity.

Additionally, the user may want to obtain her emotional state while theapparatus in in the communication mode. In operation 707, the user mayselect the FUVI icon from among the icons displayed on the seconddisplay 102. That is, the user may select sixth sense functionality fromamong a plurality of available functionalities displayed on the seconddisplay 102 while the apparatus is in a communication mode. In responseto selecting to execute a sixth sense application in operation 707, theFUVI icon 607 (shown in FIG. 6) is displayed in an enabled state on thesecond display 102, in operation 708. Additionally, a start signal iscommunicated to a headset 500 (shown in FIG. 5A), in operation 708. Inresponse to the start signal of the operation 708, the headset 500 maybe paired to the apparatus 100 or it may have been paired to theapparatus 100 prior to the apparatus 100 entering the communicationmode. In operation 709, the headset 500 starts recording audio and/orlive streaming video (supporting communication contents) and capturingthe sensory data (sensory contents) via the sensors 503 (shown in FIGS.5A and 5B, for example). In operation 710, the captured supportingcommunication contents and sensory data are processed into data blocksin synchronization with each other based on world- clock time (by way ofan example and not a limitation).

According to an exemplary embodiment, the headset 500 (shown in FIG. 5A)and/or other devices (shown in FIG. 5B) may capture in real-time audioand video contents. The video contents will include environment observedby the user which may be the two displays 201 and 202. That is, thevisual contents will include an image of the user displayed in thesecond screen 603, an image of the other entity displayed in the firstscreen 602 and the image of the additional contents displayed in thethird screen 608 and additional icons 601 and 604-607. According to anexemplary embodiment, the visual data of the additional icons may bediscarded using image recognition techniques, known in the art.According to another exemplary embodiment, the images of the userdisplayed in the second screen 603 may also be discarded using imagerecognition techniques, known in the art. The captured primary contents(audio and video) that are not discarded may be recorded in data blockse.g., 5 second intervals and stored by the headset 500 or transmitted tobe stored to the apparatus 100 or an external server. The capturedprimary contents are synchronized with the sensory data. That is,sensory signals output by the sensors 503 for a predetermined timeperiod e.g., 2.5 seconds interval, are captured and stored to determinethe emotional state of the user. In an exemplary embodiment, the sensorysignals may be stored in synchronization with the primary contents toform the data blocks. The headset 500 or the apparatus 100 maydetermine, in real-time, the emotional state of the user for arespective data block. The emotional state may then be displayed on oneof the displays.

The data blocks are generated and stored by the headset 500 and/or theapparatus 100, for further analysis, as explained in further detailbelow. The determined emotional state is output in real time on one ofthe displays of the apparatus 100. The determined emotional state may beshared with the other entity or may remain private to be visible only tothe user of the apparatus.

In an exemplary embodiment, the user may further select to displayenvironment observed by the user and the environment observed by theuser may then be displayed in real time in the third screen in a form ofan image or video.

According to an exemplary embodiment, primary data may be constructedinto an episodic content scaffolds which may be embedded extendingcontents later. FIG. 8 is a block diagram illustrating components ofgenerating data blocks according to an exemplary embodiment. Asexplained above in FIG. 5C, one or more processors such as the processorof the headset 500 together with the processor of the apparatus 100and/or a processor of an external server(s) may generate data blocksaccording to an exemplary embodiment.

The processor 800 includes a communicator 801 configured to receivecaptured primary data, which may be audio and/or video and/or additionalcontents, as explained above and also configured to receive sensory dataobtained from the sensors on a headset 500, as also explained above. Theprocessor 800 may further include a data block generator 802, which isconfigured to generate data blocks. The data block generator 802 mayinclude a filter 821, which discards portions of the primary data. Forexample, the filter 821 may discard the video data of the user of theapparatus 100. The filter 821 may also discard additional data based onuser settings such as additional contents. The data block generatorfurther includes a synchronizer 822, which synchronizes the remainingportion of the primary contents (as filtered by the filter 821) withsensory contents based on time to form a data block. For example, eachgenerated data block includes the filtered primary data and the sensorydata at time period t0-t1. The processor 800 may further include adeterminer 803. The determiner 803 determines the emotional state of theuser for each of the generated data blocks, as explained with referenceto FIG. 5C. The determiner 803 may provide the determined emotionalstate of the user in real time via the communicator 801 to the displaysof the apparatus 100. The determiner 803 may further provide thedetermined emotional state of the user to the data block generator to beadded into the data block and the communicator 801 may transmit theformed data blocks to a memory for the storage.

The cognitive or emotional state of the user may be determined by thedeterminer 803 based on processed sensory data for each segmentedportion of the data i.e., for each piece of contents, which is obtainedin synchronization with the recorded audio/video data (primary contents)i.e., for each data blocks. When sensors on a certain part(s) of thebrain output signals of a first amplitude (small amplitude) in one ormore frequency bands, the determiner 803 may determine that the user isexcited (confused, scared and/or concerned). On the other hand, when thesensors on another part of the brain output signals with a secondamplitude (large amplitude), the apparatus may determine that the useris relaxed. According to an exemplary embodiment, signals with highamplitude in a first frequency band and low amplitude in a secondfrequency band may indicate an excited cognitive or emotional state ofthe user, whereas signal with low amplitude in the first frequency bandand high amplitude in the second frequency band may indicate relaxedstate of the user. If all of the sensors produce signals of the firstamplitude, this may indicate that the user is asleep or not payingattention (very relaxed). This is provided by way of an example only andnot by way of a limitation.

The processor 800 further includes a speech to text converter 804. Thespeech to text converter 804 may obtain only the audio portion of theprimary contents via the communicator 801 (using an extractor, notshown) and convert the audio into text scripts. This is provided by wayof an example and not by way of a limitation. According to an exemplaryembodiment, the speech to text converter 804 may split the audio data(voice) into respective portions or chunks and convert to text, which isthen displayed as scripts 91 (shown in FIG. 9, explained below). Thatis, the primary contents (video/audio data) are split into segmentsbased on continuity of the voice signal via the speech to textconverter. For example, a hardware processor of the apparatus 100 maydetermine where a pause is made or an end of a sentence based on voiceintonation. According to an exemplary embodiment, equal length for asegment may also be set e.g., 2.5 seconds. This is provided by way of anexample only and not by way of a limitation. That is, according to anexemplary embodiment, the size of the data block may be determined bythe speech to text converter 804 and provided to the data blockgenerator 802 via a second contents generator.

According to another exemplary embodiment, as shown in FIG. 8, theconverted speech is provided to the secondary contents generator 805.The secondary contents are explained in greater detail below. Thescripts, along with other secondary contents such as notes and commentsmay be provided to the data block generator to be added as part of thedata block by the synchronizer 822. That is, the primary contents(audio/video data) is synchronized or linked with a correspondingdetermined cognitive or emotional state by the determiner 803 and(optionally) the segmented audio data which is converted into text bythe speech to text converter 804 for a display on as scripts (shown inFIG. 9). The scripts may be generated in real-time as the conversationtakes place. Each script may be displayed in a distinct manner (e.g.,color coded) so as to show the emotional state of the user for eachportion of the primary data.

FIG. 9 is a view illustrating an apparatus providing secondary contentsof a conversation and an emotion state of a user, while being in acommunication mode, according to an exemplary embodiment. According toan exemplary embodiment, based on a user input such as a swiping alongthe second display 102, the apparatus may further display secondarycontents generated by the secondary content generator 805 of FIG. 8. Thesecondary contents include the scripts 91 of a conversation between theother entity Susan and the user Marry. The scripts 91 may be provided intext segments 91 a-91 n based on the data blocks generated by the datablock generator 802 of FIG. 8. Additionally, each text segment mayindicate whether the audio portion of the primary data belongs to theprimary communication (live scripts) or the extended communication(later inputs) based on dotted line or solid line. Also, each textsegment may be color coded based on the emotional state of the user asdetermined by the determiner 803. That is, user's emotional state withrespect to each text segment may be displayed by for example displayingeach text segment in color corresponding to the emotional state of theuser as explained above.

FIG. 10 is a view illustrating an apparatus providing, in real-time, oneor more scripts or transcripts of a conversation and an emotional stateof a user, while being in a communication mode, according to anexemplary embodiment. As shown in FIG. 10, the second part 102 of theapparatus 100 has two screens 1001 and 1002. The first screen 1001 showsscripts of voice data uttered by the other entity, Susan and the secondscreen 1002 shows scripts of voice data uttered by the user Mary.According to an exemplary embodiment, each portion may be color-coded orprovided in a distinct manner to show the cognitive or emotional stateof the user of the apparatus.

As explained above, the generated data blocks of the communication whilethe apparatus is in the communication mode are recorded and stored in amemory. As a result, the user may review contents of the communicationin a review mode, according to an exemplary embodiment.

FIG. 11 is a view illustrating an apparatus providing secondary contentsincluding review contents and an emotional state of the user, whilebeing in a review mode, according to an exemplary embodiment.

As shown in FIG. 11, once the apparatus completes the conversation mode,the user of the apparatus may select a review icon 1101, by way of anexample. Since the communication mode is deactivated, the camera 203 isdepicted in the deactivated state and the video chat icon 601 disappearsfrom the display. Also, additional contents 1102 depicted during theconversation may be displayed on the first display 201. The seconddisplay 202 may display scripts of the recorded primary contents asSusan's scripts 1001 and Mary's scripts 1002 (explained above withreference to FIG. 10) and respective notes/comments can be made or theones that were previously made 1103 by the user of the apparatus. Thatis, according to an exemplary embodiment, additional secondary contents1103 may be displayed which is synchronized with each respective datablock and may be notes made by the user. Additionally, the scripts 1001and 1002 may be displayed in a manner which shows the cognitive oremotional state of the user of the device. The cognitive or emotionalstate may be the initial captured cognitive or emotional state of theuser during the conversation or may be cognitive or emotional state ofthe user during the review stage (current state) or a combination of thetwo.

In an exemplary embodiment depicted above with reference to FIGS. 6 and7, Mary is talking to her friend Susan. Her emotional state may beindicated by a color green because Mary likes the topic she is talkingabout. The color indicates her emotional state as being a mix of like(green). If it is yellowish, it may also indicate that she is excitedabout the topic. The screen may be depicted with a green color or acolor coded icon may be provided on a screen indicating an emotionalstate of Mary. This is provided by way of an example and not by way of alimitation. Other techniques such are shading, or even a text or an iconindicating

Mary's emotional state are within the scope of exemplary embodiments.

According to an exemplary embodiment, Mary's emotional state while theapparatus is in the conversation mode may be displayed with each scriptsegment 1105 a-1105 n while Mary's emotional state may be displayed witheach corresponding note segment 1106 a-1106 n. Accordingly, Mary mayobserve a change of her emotional state with respect to the primarycontents during the conversation mode of the apparatus and during thereview mode of the apparatus. For example, the user of the apparatus mayno longer be excited by the memory and may now feel relaxed.

FIGS. 12A and 12B are views illustrating a front and a back,respectfully, of an apparatus in a fully open state entering acommunication mode according to yet another exemplary embodiment. Asshown in FIG. 12A, the front 102 of the apparatus 100 is displayingreceiving a call from another entity such as the user's boyfriend, Jack,in real time according to an exemplary embodiment. FIG. 12B shows theback 101 of the apparatus 100 having a home screen with a plurality oficons for various functionalities of the apparatus 100 according to anexemplary embodiment.

According to an exemplary embodiment depicted in FIGS. 13 and 14, theapparatus is in a communication mode because the user has accepted thecall with another entity, Jack.

FIG. 13 is a view illustrating an apparatus providing an emotional stateof a user while being in a communication mode, according to yet anotherexemplary embodiment

As shown in FIG. 13, Mary is now talking with her boyfriend 1303 (theother entity is now Jack). The color 1310 indicates her emotion is nowred (love or excited). Jack can see how much Mary loves him when she istelling him a story or their memory together (if, of course, Mary allowshim to see it). In an exemplary embodiment of FIG. 13, the screen may beshaded or color coded based on her emotional state or a separate icon ora text may be output on a display indicating Mary's emotional state. Acolor key 1309 may be provided as well.

In an exemplary embodiment, for all operations of the apparatus, theFuvi icon 607 may be highlighted or emphasized indicating that theemotional state of the user is being monitored. In an exemplaryembodiment, when the apparatus is not connected and/or does not receiveemotional signals from the I SEE® headset, the icon 607 may be showngrayed it out i.e., depicted deactivated state of the device. In anexemplary embodiment, while the apparatus is operating, the emotionalstate of the user is being monitored and output onto one of thedisplays. Accordingly, the Fuvi icon 607 appears in an activated state.

FIG. 14 is a view illustrating an apparatus providing secondary contentsincluding review contents and an emotional state of the user, whilebeing in a review mode, according to yet another exemplary embodiment.In FIG. 14, the user's emotional state is determined to be love duringthe conversation mode as shown by the scripts 1405 a-1405 n and evenduring the review mode as shown by the notes 1406 a-1406 n.

FIG. 15 is a view illustrating an apparatus providing primary contents,secondary contents, and emotional state while the apparatus is in alearning mode according to yet another exemplary embodiment.

As shown in FIG. 15, the apparatus enters the learning mode according toa user input of a selection of a learning icon 1501. When the learningicon 1501 is selected, the apparatus enters a learning mode in which theprimary contents such as lecture 1502 is displayed on the first display201 along with a corresponding timeline 1503. According to an exemplaryembodiment, the primary contents are other than a conversation withanother user and instead may be recording a lecture in a classroom or anenvironment of a user. FIG. 15 displays the primary contents as being analready recorded lecture but this is provided by way of an example andnot by way of a limitation. The primary contents may be recorded in realtime and color coded based on the emotional state of the user.

According to an exemplary embodiment depicted in FIG. 15, the FUVI iconis activated and the headset 500 is initiated or turned to an activestate to record the environment and the emotional state of the user. Asshown in FIG. 15, the key 1505 indicates color for each emotional stateof the user. The lecture 1502 is being played to the user and thetimeline may be color coded 1503 based on the emotional state of theuser captured along with the lecture 1502. However, the secondarycontents 1506 may include the text scripts 1507 which are color codedsegments 1507 a-1507 n. The segments 1507 a-1507 n are color coded basedon the currently captured user state. In other words, according to anexemplary embodiment shown in FIG. 15, the display 201 may provide theemotional state captured during the initial recording of the primarycontents and the display 202 may provide emotional state in real time asthe primary contents 1502 is being reviewed. In the review mode, theapparatus 100 includes additional functionality such as searching withinthe primary contents, adding notes, deleting portions, editing, andmaking changes and so on, as shown in menu options 1508. The secondarycontents 1509 such as notes may be displayed on the second display 202according to an exemplary embodiment. The descriptions of the variousexemplary embodiments have been presented for purposes of illustration,but are not intended to be exhaustive or limited to the embodimentsdisclosed.

Many changes may be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the market place or to enableordinary skill in the art to understand the embodiments disclosedherein.

In an exemplary embodiment, the term “computer-readable medium” as usedherein refers to any medium that participates in providing instructionsto a processor for execution. A computer readable medium may be, forexample, but not limited to, an electronic, magnetic, optical,electromagnetic, infrared, or semiconductor system, apparatus, ordevice, or any suitable combination of the foregoing. More specificexamples (a non-exhaustive list) of the computer readable medium wouldinclude the following: an electrical connection having two or morewires, a portable computer diskette such as a floppy disk or a flexibledisk, magnetic tape or any other magnetic medium, a hard disk., a randomaccess memory (RAM), a read-only memory (ROM), an erasable programmableread-only memory (EPROM or Flash memory), a memory card, any othermemory chip or cartridge, an optical fiber, a portable compact discread-only memory (CD-ROM), any other optical medium, punchcards,papertape, any other physical medium with patterns of holes, or anyother medium from which a computer can read or suitable combination ofthe foregoing.

In the context of this document, a computer readable medium may be anytangible, non-transitory medium that can contain, or store a program foruse by or in connection with an instruction execution system, apparatus,or device.

Another form is signal medium and may include a propagated data signalwith computer readable program code embodied therein, for example, in abase band or as part of a carrier wave. Such a propagated signal maytake any of a variety of forms, including, but not limited to, theelectro-magnetic, optical, or any suitable combination thereof. Thesignal medium may include coaxial cables, copper wire and fiber optics,including the wires that comprise data bus. The signal medium may be anymedium that is not a computer readable storage medium and that cancommunicate, propagate, or transport a program for use by or inconnection with an instruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wire line, optical fiber cable, RF, etc. or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of theexemplary embodiments may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++, .Net or the like and conventionalprocedural programming languages. The program code may execute entirelyon the user's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. The remotecomputer may be connected to the user's computer through any type ofnetwork, including a local area network (LAN) or a wide area network(WAN), or the connection may be made to an external computer (forexample, through the Internet using an Internet Service Provider).

The computer-readable medium is just one example of a machine-readablemedium, which may carry instructions for implementing any of the methodsand/or techniques described herein. Such a medium may take many forms,including but not limited to, non-volatile media and volatile media.Non-volatile media includes, for example, optical or magnetic disks.Volatile media includes dynamic memory.

Various forms of computer readable media may be involved in carrying oneor more sequences of one or more instructions to a processor such as aCPU for execution. For example, the instructions may initially becarried on a magnetic disk from a remote computer. Alternatively, aremote computer can load the instructions into its dynamic memory andsend the instructions over a telephone line using a modem. A modem localto a computer system can receive the data on the telephone line and usean infra-red transmitter to convert the data to an infra-red signal. Aninfra-red detector can receive the data carried in the infra-red signaland appropriate circuitry can place the data on the data bus. The buscarries the data to the volatile storage, from which processor retrievesand executes the instructions. The instructions received by the volatilememory may optionally be stored on persistent storage device eitherbefore or after execution by a processor. The instructions may also bedownloaded into the computer platform via Internet using a variety ofnetwork data communication protocols well known in the art.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousexemplary embodiments. In this regard, each block in the flowchart orblock diagrams may represent a module, segment, or portion of code,which comprises one or more executable instructions for implementing thespecified logical functions. It should also be noted that, in somealternative implementations, the functions noted in the block may occurout of the order noted in the figures. For example, two blocks shown insuccession may, in fact, be executed substantially concurrently, or twoblocks may sometimes be executed in the reverse order, depending uponthe functionality involved. It will also be noted that each block of theblock diagram and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts, or combinations of special purpose hardware andcomputer instructions.

The terminology as used herein is for the purpose of describingparticular embodiments only and is not intended to be limiting. As usedherein, the singular forms “a”, “an” and “the” are intended to includethe plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or acts for performing the function incombination with other claimed elements as specifically claimed.

The description of the exemplary embodiments has been presented forpurposes of illustration and description, but is not intended to beexhaustive or limiting in any form. Many modifications and variationswill be apparent to those of ordinary skill in the art without departingfrom the scope and spirit of the invention. Exemplary embodiments werechosen and described in order to explain operations and the practicalapplications thereof, and to enable others of ordinary skill in the artto understand various embodiments with various modifications as aresuited to the particular use contemplated. That is, variousmodifications to these embodiments will be readily apparent to thoseskilled in the art, and the generic principles and specific examplesdefined herein may be applied to other embodiments without the use ofinventive faculty. For example, some or all of the features of thedifferent embodiments discussed above may be combined into a singleembodiment. Conversely, some of the features of a single embodimentdiscussed above may be deleted from the embodiment. Therefore, thepresent disclosure is not intended to be limited to exemplaryembodiments described herein but is to be accorded the widest scope asdefined by the features of the claims and equivalents thereof.

What is claimed is:
 1. A cognitive assistant communication apparatuscomprising: at least one communication interface configured to transmitand receive multimedia data of the communication; a first display; asecond display smaller in size than the first display; at least oneinternal camera positioned on same side as the second display; and acontroller configured to: control the at least one communicationinterface, the first display, the second display, and the camera,control the second display to display at least one participant in thecommunication and a user of the communication apparatus including acognitive state of the user, and control the first display to displaycontent of the communication obtained from the multimedia data.
 2. Thecommunication apparatus of claim 1, wherein the cognitive state of theuser is detected by a first remote apparatus and is processed by asecond remote apparatus and wherein the at least one communicationinterface is configured to communicate with the first remote apparatusvia a wireless short range communication network and to communicate withthe second remote apparatus via a communication network.
 3. Thecommunication apparatus of claim 2, wherein the content of thecommunication is converted into a text format by the second remoteapparatus and provided to the at least one communication interface andwherein the cognitive state of the user is converted into color codes bythe second remote apparatus and is displayed on the second display. 4.The communication apparatus of claim 3, wherein the first display isconfigured to display the content of the communication in the textformat and highlighted with the color codes.
 5. The communicationapparatus of claim 1, further comprising: at least one user interfaceconfigured to receive user input related to whether to detect thecognitive state of the user.
 6. The communication apparatus of claim 5,wherein the at least one user interface is a touch screen provided on atleast one of the first display and the second display.
 7. Thecommunication apparatus of claim 1, wherein the cognitive state of theuser is displayed on the second display with a cognitive state key mapindicating codes for different cognitive states of the user and whereinthe cognitive states include excited state, liking state, and a relaxstate.
 8. The communication apparatus of claim 1, wherein the controlleris configured to display on the second display the cognitive state ofthe user in correlation with the content of the communication displayedon the first display and wherein the communication includes a firstcommunication data provided by the at least one participant and a secondcommunication data provided by the user.
 9. The communication apparatusof claim 1, wherein the camera is configured to capture the user duringthe communication when the communication apparatus is open atapproximately 120 degrees and is configured to capture an environment ofthe user when the communication apparatus is open more than 180 degrees.10. A method of providing cognitive assistance comprising: establishing,by a first communication apparatus operated by a user, a communicationwith a second communication apparatus of a participant in thecommunication; receiving, by the first communication apparatus, firstmultimedia data generated by the second communication apparatus duringthe established communication; capturing, via an internal camera of thefirst communication apparatus, second multimedia data generated by theuser during the established communication; displaying, on a firstdisplay of the first communication apparatus, content of the firstmultimedia data, the second multimedia data, or a third multimedia datagenerated during the established communication, the third multimediadata includes an environment of the user or the at least one participateor data downloaded via a network; displaying on a second display of thefirst communication apparatus, the first multimedia data including theparticipant and the second multimedia data including the user; anddisplaying cognitive state of the user on at least one from among thefirst display and the second display.
 11. The method of claim 10,further comprising: receiving, by the first communication apparatus, thecognitive state of the user, wherein the cognitive state of the user isdetected by a first remote apparatus and is processed by a second remoteapparatus, and wherein the first communication apparatus is configuredto communicate with the first remote apparatus via a wireless shortrange communication network and to communicate with the second remoteapparatus via the network.
 12. The method of claim 11, wherein thecontent of the communication is converted into a text format by thesecond remote apparatus and provided to the first communicationapparatus and wherein the cognitive state of the user is converted intocolor codes by the second remote apparatus and is displayed on thesecond display of the first communication apparatus.
 13. The method ofclaim 12, wherein displaying the content includes displaying, on thefirst display of the first communication apparatus, the content of thecommunication in the text format, highlighted with the color codesobtained based on the cognitive state of the user.
 14. The method ofclaim 10, further comprising: receiving, via a user interface of thefirst communication apparatus, user input related to whether to detectthe cognitive state of the user.
 15. The method of claim 14, wherein theuser interface is a touch screen provided on at least one from among thefirst display and the second display.
 16. The method of claim 10,wherein the cognitive state of the user is displayed on the seconddisplay with a cognitive state key map indicating codes for differentcognitive states of the user and wherein the cognitive states includeexcited state, liking state, and a relax state.
 17. The method of claim10, wherein the internal camera is configured to capture the user duringthe communication when the first communication apparatus is open atapproximately 120 degrees and is configured to capture an environment ofthe user when the first communication apparatus is open more than 180degrees.
 18. A non-transitory computer readable medium configured tostore instructions, which when executed by the processor cause theprocessor to execute the following operations: connecting a firstcommunication apparatus operated by a user to establish communicationwith a second communication apparatus of a participant; obtaining, bythe first communication apparatus, first multimedia data generated bythe second communication apparatus during the communication; capturing,via an internal camera of the first communication apparatus, secondmultimedia data generated by the user during the communication;displaying, on a first display of the first communication apparatus,content of the first multimedia data, the second multimedia data, or athird multimedia data generated during the established communication,wherein the third multimedia data includes an environment of the user orthe at least one participate or data downloaded via a network;displaying on a second display of the first communication apparatus, thefirst multimedia data including the participant and the secondmultimedia data including the user; and displaying cognitive state ofthe user on at least one from among the first display and the seconddisplay.
 19. The non-transitory computer readable medium of claim 18,wherein the instructions further cause the processor to execute thefollowing operation: obtaining, by the first communication apparatus,the cognitive state of the user, wherein the cognitive state of the useris detected by a first remote apparatus and is processed by a secondremote apparatus, and wherein the first communication apparatus isconfigured to communicate with the first remote apparatus via a wirelessshort range communication network and to communicate with the secondremote apparatus via the network.
 20. The non-transitory computerreadable medium of claim 19, wherein the content of the communication isconverted into a text format by the second remote apparatus and providedto the first communication apparatus and wherein the cognitive state ofthe user is converted into color codes by the second remote apparatusand is displayed on the second display of the first communicationapparatus.